.\"  -*- nroff -*-
.\"
.\" s3backer - FUSE-based single file backing store via Amazon S3
.\" 
.\" Copyright 2008 Archie L. Cobbs <archie@dellroad.org>
.\" 
.\" This program is free software; you can redistribute it and/or
.\" modify it under the terms of the GNU General Public License
.\" as published by the Free Software Foundation; either version 2
.\" of the License, or (at your option) any later version.
.\" 
.\" This program is distributed in the hope that it will be useful,
.\" but WITHOUT ANY WARRANTY; without even the implied warranty of
.\" MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
.\" GNU General Public License for more details.
.\" 
.\" You should have received a copy of the GNU General Public License
.\" along with this program; if not, write to the Free Software
.\" Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
.\" 02110-1301, USA.
.\"
.\" $Id$
.\"
.Dd June 21, 2008
.Dt S3BACKER 1
.Os
.Sh NAME
.Nm s3backer
.Nd FUSE-based single file backing store via Amazon S3
.Sh SYNOPSIS
.Nm s3backer
.Bk -words
.Op options
.Ar bucket
.Ar /mount/point
.Ek
.Sh DESCRIPTION
.Nm
is a filesystem that contains a single file backed by the Amazon Simple Storage Service (Amazon S3).
As a filesystem, it is quite small and simple: it provides a single normal file having a fixed size.
The file is divided up into blocks, and the content of each block is stored in a unique Amazon S3 object.
.Pp
In typical usage, a `normal' filesystem is mounted on top of the file exported by the
.Nm
filesystem using a loopback mount.
This arrangement has several benefits:
.Bl -tag -width xx
.It o
You get full UNIX filesystem semantics and can configure the `upper'
filesystem any way you want.
.It o
When storing your data on Amazon S3 servers, which are not under your control, encrypting the data
becomes important. This capability is supported natively on Linux via loopback mounts.
.It o
Since S3 data is accessed over the network, local caching is also very important for performance reasons.
By making the
.Nm
filesystem block size equal to (or a multiple of) the kernel's native block size, all of the
filesystem caching can be done where it should be: in the kernel via the kernel's page cache.
.Nm
itself does not cache any file data.
.It o
The user benefits from the ability to create a multi-terabyte filesystem stored in the `cloud'
and available from anywhere on the Internet.
.El
.Pp
As a simple optimization,
.Nm
does not store blocks containing all zeroes; instead, they are simply deleted.
Conversely, reads of non-existent blocks will contain all zeroes.
Therefore, no initialization is necessary when creating a new filesystem.
.Pp
Since Amazon S3 makes relatively weak guarantees relating to the timing and consistency of reads vs. writes
(collectively known as ``eventual consistency''),
.Nm
includes logic and configuration parameters to work around these limitations.
These are:
.Bl -tag -width xx
.It 1.
Regarding PUT/DELETE operations on the same block: avoidance of overlapping operations plus a configurable
minimum delay between consecutive operations. This ensures S3 doesn't receive these operations out of order.
.It 2.
An internal MD5 checksum cache, which enables
.Nm
to automatically detect and reject `stale' information returned by GET operations.
.El
.Pp
Althogh it functions over the network, the
.Nm
filesystem is not a distributed filesystem and does not support simultaneous mounts.
.Sh OPTIONS
Options specific to
.Nm
are as follows:
.Bl -tag -width Ds
.It Fl \-accessFile=FILE
Specify a file containing `accessID:accessKey' pairs, one per-line.
Blank lines and lines beginning with a `#' are ignored.
If no
.Fl \-accessKey
is specified, this file will be searched for the entry matching the access ID specified via
.Fl \-accessId;
if neither
.Fl \-accessKey
nor
.Fl \-accessId
is specified, the first entry in this file will be used.
Default value is
.Pa $HOME/.s3backer_passwd .
.It Fl \-accessId=ID
Specify Amazon S3 access ID.
If no access ID is specified (and none is found in the access file) then
.Nm
will still function, but only reads of publicly available filesystems will work.
.It Fl \-accessKey=KEY
Specify Amazon S3 access key. To avoid publicizing this secret via the command line, use
.Fl \-accessFile
instead of this flag.
.It Fl \-accessType=TYPE
Specify the Amazon S3 access privilege ACL type for newly written blocks.
The value must be one of `private', `public-read', `public-read-write', or `authenticated-read'.
.It Fl \-baseURL=URL
Specify the base URL. Default is `http://s3.amazonaws.com/'.
.It Fl \-blockSize=SIZE
Specify the block size. This must be a power of two and should be a multiple of the native page size.
The size may have an optional suffix 'K' for kilobytes, 'M' for megabytes, etc.
.Nm
supports partial block operations, though for writes this may require a read and then a write.
Proper alignment of the
.Nm
block size with the intended use (e.g., the block size of the `upper' filesystem) will minimize or
eliminate the extra reads.
.Nm
will attempt to auto-detect the block size by reading block number zero.
If this option is not specified, the auto-detected value will be used.
If this option is specified but disagrees with the auto-detected value,
.Nm
will exit with an error unless
.Fl \-force
is also given.
If auto-detection fails because block number zero does not exist, and this option is not specified,
then the default value of 4K (4096) is used.
.It Fl \-cacheSize=SIZE
Specify the size of the MD5 checksum cache (in blocks).
If the cache is full when a new block is written, the write will block until there is room.
Therefore, it is important to configure
.Fl \-cacheTime
and
.Fl \-cacheSize
according to the frequency of writes to the filesystem overall and to the same block repeatedly.
Alternately, a value equal to the number of blocks in the filesystem eliminates this problem but consumes
the most memory when full (each entry in the cache is approximately 40 bytes).
A value of zero disables the cache.
Default value is 1000.
.It Fl \-cacheTime=MILLIS
Specify in milliseconds the time after a block has been successfully written for which the MD5 checksum
of the block's contents should be cached, for the purpose of detecting stale data during subsequent reads.
A value of zero means `infinite' and provides a guarantee against reading stale data; however,
you should only do this when
.Fl \-cacheSize
is configured to be equal to the number of blocks; otherwise deadlock will (eventually) occur.
This value must be at least as big as
.Fl \-minWriteDelay.
This value must be set to zero when
.Fl \-cacheSize
is set to zero (cache disabled).
Default value is 10 seconds.
.It Fl \-connectTimeout=SECONDS
Specify a timeout in seconds for the initial HTTP connection.
Default is 30 seconds.
.It Fl \-filename=NAME
Specify the name of the single file that appears in the
.Nm
filesystem.
Default is `file'.
.It Fl \-force
Proceed even if the value specified by
.Fl \-blockSize
or
.Fl \-size
disagrees with the auto-detected value.
This is will certainly lead to reading garbled data and should only be used when you
intend to write over an existing filesystem with a new one.
.It Fl h Fl \-help
Print a help message and exit.
.It Fl \-ioTimeout=SECONDS
Specify a timeout in seconds for the completion of an HTTP operation after the initial connection.
Default is 30 seconds.
.It Fl \-maxRetry=COUNT
Specify the number of times
.Nm
should retry the HTTP operation when it fails.
Failures include network failures and timeouts, 5xx server errors, and reads of stale data
(i.e., MD5 mismatch).
If the last attempt fails, an error will be returned.
Default value is 9 retries (10 total attempts).
.It Fl \-retryPause=MILLIS
Time in milliseconds to pause between the aforementioned retries.
Default value is 1000ms (1 second).
.It Fl \-minWriteDelay=MILLIS
Specify a minimum time in milliseconds between the successful completion of a write and the initiation
of another write to the same block. This delay ensures that S3 doesn't receive the writes
out of order.
Default value is 500ms.
.It Fl \-prefix=STRING
Specify a prefix to prepend to the resource names within bucket that identify each block.
Default is the empty string.
.It Fl \-size=SIZE
Specify the size (in bytes) of the single file to be exported by the filesystem.
The size may have an optional suffix 'K' for kilobytes, 'M' for megabytes, 'G' for gigabytes, or 'T' for terabytes.
.Nm
will attempt to auto-detect the block size by reading block number zero.
If this option is not specified, the auto-detected value will be used.
If this option is specified but disagrees with the auto-detected value,
.Nm
will exit with an error unless
.Fl \-force
is also given.
.It Fl \-version
Output version and exit.
.El
.Pp
In addition,
.Nm
accepts all of the generic FUSE options as well.
Here is a partial list:
.Bl -tag -width Ds
.It Fl d
Enable debug mode (implies
.Fl f ) .
.It Fl f
Run in the foreground (do not fork).
.It Fl s
Run in single-threaded mode.
.It Fl o Ar allow_root
Allow root (only) to view backed file.
.It Fl o Ar allow_other
Allow all users to view backed file.
.It Fl o Ar nonempty
Allow all users to view backed file.
.It Fl o Ar uid=UID
Override the user ID of the backed file, which defaults to the current user ID.
.It Fl o Ar gid=GID
Override the group ID of the backed file, which defaults to the current group ID.
.It Fl o Ar sync_read
Do synchronous reads.
.It Fl o Ar max_readahead=NUM
Set maximum read-ahead (in bytes).
.El
.Sh FILES
.Bl -tag -compact -width Ds
.It Pa $HOME/.s3backer_passwd
Contains Amazon S3 `accessID:accessKey' pairs.
.El
.Sh SEE ALSO
.Xr mount 8 ,
.Xr umount 8 ,
.Xr fusermount 8 .
.Rs
.%T "s3backer: FUSE-based single file backing store via Amazon S3"
.%O http://s3backer.googlecode.com/
.Re
.Rs
.%T "Amazon Simple Storage Service (Amazon S3)"
.%O http://aws.amazon.com/s3
.Re
.Rs
.%T "FUSE: Filesystem in Userspace"
.%O http://fuse.sourceforge.net/
.Re
.Rs
.%T "Google Search for `linux page cache'"
.%O http://www.google.com/search?q=linux+page+cache
.Re
.Sh AUTHOR
.An Archie L. Cobbs Aq archie@dellroad.org
